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©cample, we can survey project managers about on-going projects 
and their characteristics in order to predict how much effort they 
will require. We can correlate the predictions with the outcomes 
to assess predictive criterion validity. 

Construct validity 

Construct validity concerns how an instrument "behaves" when 
used. We can think of this behavior in two ways. Convergent 
construct validity assesses the extent to which different data 
collection approaches produce similar results. Divergent 
construct validity assesses the extent to which results do not 
correlate with similar but distinct concepts. It often requires 
many years of experience to assess construct validity properly. 

Validity and Reliability in Software Engineering 
Surveys 

Generally, software engineering surveys are weak in the area of 
validity and reliability. For example, for many years, in the 
extensive literature relating to the CMM, there was only one 
reference to a reliability coefficient (the Cronbach's alpha) and 
that concerned the 1987 version of the Maturity Questionnaire 
(5], Of the three surveys we discussed in part 1 of this series, 
only the Finnish Survey [8] made a concerted effort to undertake 
reliability and validity studies. Our own studies of technology 
adoption used face validity only. Lethbridge [6] and [7] discusses 
the basis for his questions, but his discussion of validity is based 
only on a post-hoc assessment of possible responder bias. 

In contrast, the Finnish researchers used a panel of experts to 
judge the content validity of the questions. They also attempted to 
assess the internal reliability of their instrument. Unfortunately, 
they did not perform an independent pilot study. They analyzed 
their survey resp on ses using principal components to identify 
strategies for managing risks. They then derived Cronbach alpha 
statistics [2] from the same responses. They found high values 
and concluded that their survey instrument had good reliability. 
However, Cronbach alpha values were bound to be high, because 
they measure the structure already detected by the principal 
component analysis. 

Survey Documentation 

After the instrument is finalized, Bourke and Fielder [1] 
recommend starting to document the survey. If the survey is self- 
administered you should consider writing an initial descriptive 
document, called a questionnaire specification. It should include: 

• The 6bjective(s) of the study. 

• A description the rationale for each question. 

• The rationale for any questions adopted or adapted from 
other sources, with appropriate citations. 

• A description of the evaluation process. 

Furthermore, once the questionnaire is administered, the 
documentation should be updated to record information about: 

• Who the respondents were. 

• How it was administered 

• How the follow-up procedure was conducted 



♦ How completed questionnaires were processed. 

One of the major reasons for preparing documentation during the 
survey is that surveys can take a long time. It may be many 
months between first distributing a questionnaire and when we 
are able to analyze results. It takes time for respondents to reply 
and for the researchers to undertake all necessary follow-up 
procedures. This time lag means that it is easy to forget the 
details of instrument creation and administration, especially if 
documentation is left to the end of the study. In general, it is 
good research practice to keep an experimental diary or log book 
for any type of empirical studies 

When questionnaires are administered by interview, 
specifications are referred to as interviewer specifications and 
can be used to train interviewers as well as for reference in the 
field. 

Once all possible responses have been received and all follow-up 
actions have been completed, we are in a position to analyze the 
survey data. In the next two articles, we discuss survey sampling 
. and analysis issues. 
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